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INPUT METHOD, SYSTEM AND DEVICE 

Field of the Invention 

The present invention relates generally a method for entering input into a 
5 computing system and an input device for use with a computing system. More 
particularly, the invention relates to methods and apparatus for sensing tactile 
input. 

Background to the Invention 

10 Many prior art methods exist for entering input into a computing system. 

The most widely accepted methods involve the use of a keyboard to input 
character data. A mouse is often used in conjunction with a keyboard to 
manipulate the position of a cursor on the screen by translating movement of 
the mouse across a substantially flat surface into a corresponding movement of 

1 5 the cursor on screen. 

Touch screens have become increasingly popular as input devices, 
particularly for non technical applications such as public information displays. 
Touch screens include a touch sensor panel placed over a display screen so 
that the touch sensitive area of the sensor panel covers the viewable area of the 

20 display. Various types of touch sensor systems are available based on 
resistance, capacitance or ultrasonic methods to detect input by touch. A sensor 
panel typically includes an electrical current passing over it such that touching 
the screen causes a change in electrical signal. The change in electrical signal 
is used to determine the position of the touch on the display screen. 

25 Graphics tablets are a form of computer input device which operates in 

conjunction with a stylus for drawing and editing images or for performing 
typical mouse-type functions. Graphics tablets are particularly suited to 
drafting, sketching and so forth. Other computing system input devices such as 
rollers, thumbwheels and trackballs are useful for panning and scrolling 

30 functions. 

However, each prior art input method is typically suited to a specific type 
of input and becomes awkward and inefficient to use when employed to enter 
other types of input into a computing system. For instance, whilst a keyboard 
offers superior efficiency for typing applications, it is not particularly suitable for 
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manipulation of graphic objects and the like. For both technical and non 
technical computing system operators, the variety of tasks that a single operator 
may encounter over a single day may call for a number of input devices. This 
leads to the inconvenience of switching between input devices and the 
5 disadvantage of having a number of input devices connected to a computing 
system. In addition, it is recognised that many existing input devices have not 
been designed with ergonomics in mind, making them awkward to use and 
predisposing users to problems including repetitive strain injuries. 

Some prior art devices and systems exist which operate via tactile 

10 sensing means to provide data input to a computing system. These devices 
typically use finger positioning on a capacitance, resistance or ultrasonic touch 
pad in place of other input devices. Changes in conditions caused by a finger 
touching the touch pad are detected by sensors. By sensing the position of a 
finger at consecutive intervals, the motion of the finger can be monitored. Such 

15 sensing devices may be used to manipulate movement of a cursor on a 
computer screen. However, such prior art tactile sensing devices have not 
been widely accepted. Touch sensitive keyboards tend to be harder to use and 
produce more errors. The limited range of movement and resolution of pointing 
type devices often leads to slower and less accurate output than individual input 

20 devices. 

A number of integrated input devices also exist which combine 
functionalities such as keyboard and mouse into a single device. However, 
integration of a number of functionalities irito a single input device may add an 
undesirable level of mechanical complexity to the device. 
25 The discussion of the background to the invention included herein is 

included to explain the context of the invention. This is not to be taken as an 
admission that any of the systems referred to were published, known or part of 
the common general knowledge as at the priority date of the claims. 

30 Summary of the Invention 

According to a first aspect of the present invention, there is provided a 
m e thod of e nt e ring i nput into a computing syst e m i ncluding th e st e ps of: 

fa) detect i ng input mov e m e nts by m o ans of a panel inc l uding an array 

of s e nso rs; 
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(b) proc e ssing — the — d e tected — input — mov e ments — by — consulting — a 

know le dge database — to identify a corr e sponding instruct i on s i gnal for each 
det e cted i nput movem e nt; and 

(€) transm i tt i ng the instruction signal to th e computing syst e m. 

5 method of entering input into a computing system, the method comprising the 
following steps: 

detecting one or multiple input movements using a sensing panel 

associated with the computing system; 

classifying each detected input movement as being of a particular type; 

10 translating each input movement to an instruction signal by consulting a 

knowledge database; and 

transmitting the instruction signal to the computing system; 

wherein translation of a detected input movement to an instruction signal 

involves a main process and one or more sub-processes, wherein each sub 
15 process is invoked by the main process in response to a particular type of 
detected input movement. 



The sensing panel comprises an array of sensors for generating images 
of a contact area for receiving input movements. The input movements may be 

20 executed by a user's finger or fingers, or any suitable peripheral device such as 
a pen, stylus or any other implement whose movement may be sensed by the 
panel. The sensing panel detects changes in the position of a user's finger or a 
peripheral device between images. Processing of the input movements is 
preformed by a processor and involves consulting a knowledge database to 

25 translate the detected input movements into corresponding instruction signals 
which have been previously mapped to the knowledge database. 

In a preferred embodiment of the present invention, each 

particular type of input movements is associated with operation of the sensing 
panel in a different mode. That is, singular pecking movement which simulates 

30 typing a key may be associated with operation of the device in a keyboard 
mode, whilst a sliding motion may be associated with a cursor mode or a mouse 
mode. Other types of modes may include scripting modes which may be used 
for freehand writing or drawing and the like as well as for sending commands, 
device modes which operate in association with peripheral devices, customer 
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modes which comprise user defined modes and idle modes. Whilst a main 
process may manage any number of sub processes, each sub process is 
assigned to translation of an input movement into an instruction signal 
th o processing st e p involv e s using a hi e rarchical control structur e including one 
5 or more sub procoss o s to be i nvok e d by a ma i n procoss i n rosponso to a 
particular type of d e tect e d input mov e ment such that th e sub proc e ss can 
associat e the d e t e ct e d input mov e m e nts with an appropriat e instruction signal 

According to an embodiment, the main process and one or more sub 

processes together form a hierarchical control structure in which the main 
10 process determines whether an input movement corresponds to a prompt to 
invoke a particular mode, and where a particular mode is indicated, the main 
process invokes a sub process in that mode. 

The term "Process" as used in this specification refers to any execution 

procedure including a thread or a task and is not limited to "an instance of a 

15 program running in a computer" or any other limited definition which may be 
used in computer science. 

Particu l ar input mov e ments may be associat e d with op e ration of th e 
sensing pan el in diff e r e nt mod e s. — That is, singular pecking mov e m e nt wh i ch 
simulates typing a k e y may b e associ a ted with op e ration of th e dev i ce in a 

20 keyboard mod e , wh i lst a s l iding motion may be associated with a cursor mod e 
or a mous e mod e . — Other typ e s of mod e s may i nclude script i ng mod e s wh i ch 
may b e us e d for fr e ehand writing or drawing and the l i k e as w ell a s for s e nding 
eefRm af*d s , de v i c e modes-whi ch op e ra te i n association w i th p ef-ipberal de v ices, 
custom e r mod es which cempr-ise-user d e fin e d mod e s and idle modes. W hi ls t a 

25 main process may manag e any numb e r of sub process e s, each sub proc e ss is 
assigned to trans l ation of a n input movem e nt into an instruction signal. 

In a preferred form of the invention, the main process manages one or more 

sub processes by assigning a priority value such that a sub process having a minor 
priority value does not impede a sub process having a major priority value. 

30 th e main proc e ss ass i gns a prior i ty value to e ach sub process invoked via a 
registrat i on proc e ss th e reby e nsuring th a t a sub process having a minor pr i ority 
valu e do e s not imp e de a sub proc o ss having a major pr i ority va l ue. 

Each currently operating sub process is assigned a priority value to avoid 

interference between conflicting sub processes. Priority values may be assigned 
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according to factors including the order in which the sub processes were requested 
and the mode associated for the sub process. 

Preferably, each invoked sub process claims a region of the sensing 
panel such that any input movements received via the claimed region of the 
5 sensing panel will be translated by the sub process having claimed the region, a 
sub proc e ss claims a region of the pan el th e r e by causing all input mov e m e nts 
r e ceived via th e r e gion to be processed by that sub process. More pr e f e rably, 
more than one sub proc e ss c l aims th e s a m e or part of th e sam e r e g i on of th e 
pane l such that the regions c l aimed by ind e pend e nt sub proc e ss ov e r l ap. 

10 Each sub proc e ss c l aims a r e g i on of the s e ns i ng pan e l as its working 
ar e a which usually corr e sponds to a reg i on with i n th e imag e s gen e rat e d by th e 
array of s e nsors. — Wh e r e th e claim e d r e gions of diff e r e nt sub proc e ss e s 
o verlap, th e sub proc e ss e s p r oc e ed on th e basis of th ei r priority values: 
More preferably each input received via a region having been claimed by 

15 a sub process translated only by the sub process having claimed that region of 
the sensing panel, or by a sub process having a higher priority value than the 
sub process having claimed that region of the sensing panel- 
Once the claiming sub process is complete, the claimed region may revert to an 
unclaimed status. 

20 

According to a second aspect of the present invention, there is provided an 
input system for a computing system, the input device comprising: 

a sensing panel including an array of sensors for detecting input 

movements; 

25 a processor for classifying each detected input movement as being of a 

particular type and translating each input movement to an instruction signal for 
transmission to the computing system; and 

a knowledge database for consultation by the processor to identify the 

instruction signal corresponding to the detected input movement: 

30 wherein translation of a detected input movement to an instruction signal 

involves a main process and one or more sub-processes, wherein each sub 
process is invoked by the main process in response to a particular type of 
detected input movement. 
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(a) a pane l including an array of s e nsors for d e t e ct i ng — input 

movem e nts; 

(&) a processor for proc e ssing th e d e t e ct e d input mov e ments by 

consulting a know l edge database to i d e nt i fy a correspond i ng instruction signal 
5 for each detected input movomont; and 

(g) a s i gnal transmission means for transmitting the instruction s i gnals 

to t ho c om putin g system. 

A cco r ding to a thi rd as pe ct of th e i nv e nt i on, th e re is provided an i n put 
devic e for use with a computing syst e m, the input device inc l uding: 

10 (a) a panel including an array of s e nsors for d o t o cting input 

movements; 

(b) m e ans of transm i tting the d o t o ct e d input mov o m o nts to a 

processor for proc e ssing by consu l ting a knowl e dge database to id e ntify a 
corrosponding — i nst ruc tion — signal — far — e ach — d et e ct e d — i nput — move m e nt — for 

1 5 transmiss io n to th e c omp u ti fi^-svstem TPreferablv, the sensors for detecting 
input movements are light detecting sensors. The array of sensors record light 
falling on them and generate images of the panel providing the contact surface 
for the input device. The sensors generate images of the contact surface and 
detect changes between images recorded in seguence which are identified as 

20 input movements. The light detecting sensors may comprise any suitable 
sensor such as a charge-coupled device (CCD) array but is most preferably an 
array of complementary metal oxide semiconductor sensors (CMOS). 

The sensing panel may further include a display layer for guiding user 

input. 

25 The knowledge database may be dynamic, allowing an operator to 

change or redefine the database to fit individual circumstances. The knowledge 
database may interact with external applications and can be modified externally 
by host computing system, independent application software and external 
devices. The data in the knowledge database may be supplied partially or 

30 completely by the computing system, independent application software and/or 
external devices, so that the knowledge database is customised according to 



the needs of the computing system, independent application software and/or 



external devices. The customised knowledge database may form an integral 
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part of the computing system, independent application software and/or external 
devices. 

According to another embodiment of the invention, there is provided an 

input device for use with a computing system for entering input into the 
5 computing system, the input device comprising a sensing panel and a 
transmission component for transmitting detected input movements to a 
processor to identify an instruction signal corresponding to the detected input 
movement in accordance with the method described. 

The input device is able to connect to connection ports such as serial 

10 ports for connecting a keyboard or mouse. The device components may be 
integrated into a single mechanical device, or alternatively, may be hosted on 
another device, such as the computing system to which they are providing a 
means of entering input. For instance, the sensing panel could transmit input 
movements to a processor associated with the computing system to enable a 

1 5 sharing of resources. 

In a pr e ferr e d e mbodim e nt of th e pres e nt inv e ntion, t The input device 
may further includes computer software which causes the processor to operate 
using a hierarchical control structure including one or more sub processes to be 
invoked in response to a particular type of detected input movements such that 

20 the sub process can associate the detected input movements with an 
appropriate instruction signal. 

Pr o f o rably, the sensors for d e t e cting input mov e m e nts ar e light det e cting 
se nsors. The arr a y of s e nsors r e cord light falling on th e m and gen e rat e imag e s 
Q^4ho panel provi di ng th e c ofrta ct surf a c e for tho input d e vic e . — The sensors 

25 g e n e rate imag e s of the contact surfac e and detect changes b e tw e en i mag e s 
recorded in s e qu e nc e which are id e ntif i ed as input movem e nts. — The light 
d e t e cting s e nsors may comprise a ny suitab l e sensor such as a charg e coup le d 
device (CCD) array but is most pref e rably an array of complem e ntary m e tal 
oxid e s e miconductor s e nsors (CMOS). 

30 I n on e form of th e inv e ntion, th e input mov e m e nts d e t ec t ed a re the 

m ovem e nts of ono or more user fin gefs-r 

I n y o t anoth e r form of th e invention, th o input d e vic e inc l ud e s a 
mov e m e nt indicating d e v i c e , w he r e in the input mov e ment s d e t e ct e d are th e 
movements of th e movem e nt ind i cating d e vic e , the movem e nt of which across 
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th o surfac e of th e pan el indicates an instruct i on signa l to mov e in th e dir e ction 
indicat e d with the movem e nt i ndicat i ng devic e . — Pr e f e rably, an app l ication of 
pr e ssure to th e mov e ment indicating d e vice causes an input movement wh i ch is 
interpr e ted by th e proc e ssor as ind i cat i ng an instruction s i gnal to mov e 
downw a rds, and reducing th e pressure applied to th e movem e nt ind i cating 
d e vic e caus e s a n i nput mov e ment which is interpr e ted by th e proc e ssor as 

having a surfac e suitabl e for forming a contact with th e s e nsing pan e l. — One 
10 such su i tab l e form may be an e lasticised h e m i sph e rica l form which may b e 

moved across th e sensing pan el by a user's hand and may b e compressible in 

respons e to an application of pr e ssur e by th e user's hand. 

According to a fourth asp e ct of the invention, th e r e is prov i d e d a 

comput e r softwar e program for proc e ssing input for a computing syst e m, th e 
15 c omputer program including: 

(a) a m e ans for rec e iving input mov e ments from a pan el including an 

array of sensors; 

(fe) a m e ans for proc e ssing the input mov e m e nts inc l ud i ng consulting 

a k nowle d g e database to identify a corresponding instruction s i gnal for e ach 
20 input movement; and 

(e) a m e ans for transm i tt i ng th e instruction signa l s to the computing 

s yst e m for output. 

I n a pr e f erred e mb o dim ent of th e inv e ntion, th e me ans for proc e ssing the 

input mo vem e nts inv olves-a-tae rarch i c al control structur e including on e or mo pe 

25 sub process e s to be invok e d by a main proc e ss i n r e spons e to a part i cular typ e 
of det e ct e d input m o v e m e nt such that th e sub proc e ss can associate each 
d e t e ct e d input mov e m e nt w i th a n appro pr i at e instruction signa l - 
According to a f i fth aspect of th e invention, there is prov i ded a knowledg e 
databas e for use in co nju nction with an input syste m of the typ e pr e vious l y 

30 d e scribed, th e know le dg e d atabase storing i nformation r e lating to diff e rent 
mod e s of o p e rati on of th e i np ut s yst e m, diff e rent movem e nts which can b e 
detected by t he inp u t s ystem, corres ponding signa ls as s oci at ed w ith th e 

syst e m, and support i ng information such as how to assign prioriti e s to sub 



processes assoc i at e d with the proc e ss i ng of d e tect e d movements, and how to 
manage comput i ng r e sources. 

Tho know l edge databas e may b e dynam i c, al l owing a n operator to chang e or 
rodofino the databas e to fit individual circumstanc e s. The knowledg e databas e 
5 m a y int e ract with ext e rna l applications and can b e modif ie d e xt e rnally by host 
computing syst e m, ind e p e n de nt application softwar e and e xt e rnal d e vic e s. The 
data in the knowledge databas e may b e supp lie d partia l ly or comp l etely by th e 

go that th e knowl e dg e database is customis e d according to the needs of the 
10 computing syst e m, i nd e p e ndent app l icat i on software and/or externa l d e v i c e s. 
Th e customised know le dg e databas e may form an int e gral part of th e 
computing syst e m, ind e pendent appl i cation softwar e and/or e xt e rnal d e vic e s. 

Brief Description of the Drawings 

15 The invention will now be described in further detail by reference to the 

attached drawings illustrating example forms of the invention. It is to be 
understood that the particularity of the drawings does not supersede the 
generality of the preceding description of the invention. In the drawings: 

Figure 1 is a schematic representation of interactions between 

20 components of an input system, according to an embodiment of the present 
invention. 

Figure 2 illustrates a simplified structure of the sensing panel having an 
array of sensors. 

Figure 3 illustrates a flowchart of example operations of a main process. 
25 Figure 4 illustrates a flow chart of the operations of a typical sub process. 

Figure 5 illustrates a flowchart demonstrating the interaction between a 
main process and a keyboard sub process and a cursor sub process. 
Figure 6 illustrates a variation of the flowchart in Figure 5. 
Figure 7 illustrates a flowchart demonstrating the interaction between a 
30 main process and a number of sub processes. 

Figure 8 illustrates a keyboard layout according to one embodiment of 
the invention. 

Figure 9 illustrates examples of user input patterns used to provide input 
to a computing system in accordance with an embodiment of the invention. 
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Figure 10 illustrates more examples of user input patterns used to 
provide input to a computing system in accordance with an embodiment of the 
invention. 

Figure 11 illustrates more examples of user input patterns used to 
provide input to a computing system in accordance with an embodiment of the 
invention. 
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Detailed Description of the Drawings 

Figure 1 is a schematic representation of interactions between 
independent components of an the input device-4. The input device^- provides 
5 input to a computing system connected to the output means 2. A sensing panel 
3 provides a contact surface 4 (see Figure 2) to the user. The sensing panel 3 
includes an array of sensors 5 (see Figure 2) which detect input movements 
entered by a user using single or multiple fingers-© or a peripheral device such 
as a pen or stylus. The array of sensors 5 generates an image-7 of the contact 

10 surface 4 to enable detection of changes between sequentially recorded images 
7. These changes are identified as input movements. A processor 8 processes 
the images^ by consulting a knowledge database 9 to identify a corresponding 
instruction signal for each detected input movement. The instruction signals are 
transmitted to a computing system to provide appropriate output. 

15 The system components may be integrated into a single device, or 

alternatively integrated with other devices such as a computing system. For 
example, the sensing panel 3 may be connected to a computing system to 
which it provides input, whilst the processor 8 and knowledge database 9 are 
integrated with the computer system. In this type of arrangement, the processor 

20 8 shares resources with the computer system. In an alternate arrangement the 
sensing panel 3, processor 8 and knowledge database 9 may be incorporated 
into a single unit further including an output means 2, for example, a screen for 
visual output or speakers for auditory output. Where the components are 
provided as discrete units, connection means must be provided to facilitate 

25 communication between devices. 

Figure 2 illustrates a simplified structure of the sensing panel 3 including 
an array of sensors 5 for use with the ; present invention. The sensing panel 3 is 
constructed in several layers. An upper layer 10 provides a contact surface 4 to 
receive input movements entered by a user. The upper layer 10 comprises a 

30 detachable, transparent and flexible surface, which optionally has a keyboard 
layout 11 (see Figure 8) printed on its upper surface. Beneath the upper layer 
is a transparent a resilient layer 12 is provided to offer protection to underlying 
layers and to provide tactile feedback to the user. The transparent resilient 
layer 12 is designed to minimize any strain associated with the impact of typing 
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on a user's fingers-§. Furthermore, the keyboard layout 1 1 may be adapted to 
each individual users preferences, making it possible to provide an ergonomic 
typing system to suit individual user's hand sizes and input requirements. 

Beneath the transparent resilient layer 12 is an array of light sensors 5 
5 for detecting the input movements received via the contact surface 4. The light 
detecting sensors are most preferably CMOS sensors. These sensors detect 
light patterns for subsequent transformation into images-7 or electrical charge 
patterns. A circuit board^l-S located beneath the sensing layer 5, to collect 
signals from the array of sensors 5 and generate the images^? which are then 

10 processed by the processor 8. Each pixel on the resulting imaged may 
correspond to a single sensor 14, that is, the position of the pixel in the imaged 
is directly related to the position of the sensor 14 within the array of sensors 5. 
Alternatively, a single pixel on the resulting image-7 may correspond to any 
number of sensors 14. Similarly, it is to be understood that a single sensor 14 

15 could provide input corresponding to any number of pixels on the resulting 
image-7. The value of each pixel is dependent on the magnitude of the signal 
generated by the corresponding sensor 14. A back lighting source 15 is 
optionally incorporated into the sensing panel 3. In this arrangement, the light 
patterns detected by the sensors 14 will be based on reflection patterns. 

20 The placement of fingers-§ or some peripheral device on the contact 

surface 4 creates an impression~4£ on the upper layer 10 of the sensing panel 3 
comprising an input movement. An input movement causes the amount of light 
received by individual sensors 14 to vary from regions of the contact surface 4 
which do not assume the impression~~46 of a finger-6 or peripheral device. 

25 Therefore, the imaged formed in one particular frame whilst a user is entering 
an input movement, will vary from an image-? formed when no finger-6 or 
peripheral device is connecting with the contact surface 4. 

As an alternative to having the keyboard layout 1 1 printed on the sensing 
panel 3, the sensing panel 3 may further include a display layer 17. Such a 

30 display layer 17 may comprise a flexible liquid crystal display (LCD) positioned 
between the contact surface 4 and the array of sensors 5. The LCD layer 17 
may be used to display information and guide user input, such as a keyboard 
layout 11. It is to be appreciated that where a LCD 17 is used to display a 
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keyboard layout-44, a dynamic display is enabled thereby permitting the 
keyboard layouts to be redefined in response to changes in a user's needs. 

In another form of the invention not illustrated here, the sensing panel 3 
includes a detachable, opaque and flexible layer which optionally has a 
5 keyboard layout-44 printed thereon. The inner surface of the upper layer 10 is 
coloured in a uniform block colour such as black, blue or red. Beneath the 
upper layer 10 is a flexible, semi-transparent layer serving a cushioning 
function. This flexible, semi-transparent layer may comprise a liquid substance. 
In this particular arrangement, when a finger-© or peripheral device 

10 connects with the contact surface 4 and leaves an impression-46 thereon, the 
depth of the underlying liquid layer decreases directly beneath the impression 
4§, influencing the manner in which underlying sensors 14 will detect the light 
reflected by the inner surface of the upper layer 10. Furthermore, comprising 
the flexible, semi-transparent layer of liquid, enhances tactile feedback to the 

15 user. As described previously, a dynamic display means such as an LCD layer 
17 may be incorporated into the arrangement 

A processor 8 reads in the sequentially generated image frames in order 
to translate input movements entered by a user's finger-S or suitable peripheral 
device into corresponding instruction signals by consulting a knowledge 

20 database 9. The resulting instruction signals are transmitted to a computing 
system for output. Translation of input movements into instruction signals 
occurs via a combination of two types of operational processes, main processes 
18 x and sub-processes 19 (see Figure 3) creating a hierarchical control 
structure including one or more sub processes 19 to be invoked by a main 

25 process 18 in response to a particular type of detected input movement so that 
the sub process 19 can associate the detected input movements with an 
appropriate instruction signal. That is, particular input movements are 
associated with different operational modes of the sensing panel 3. For 
example, a single pecking movement simulating typing of a key-2-0 may signal a 

30 prompt for the input device 1 to operate in keyboard mode. Similarly, a sliding 
finger motion may signal a prompt for the input device 1 to operate in cursor 
mode. Main processes 18 provide for the management of sub-processes 19 
including initiating sub processes, allocating system resources to sub 
processes, monitoring of sub processes and collecting the instruction signals 
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translated by the sub processes. Each sub process 19 is assigned to the 
translation of one or more particular input movements into corresponding 
instruction signals by consulting the knowledge database 9. 

The knowledge database 9 is effectively a database including a bank of 
5 stored input movements; information regarding the various modes of operation 
of the input device 1 including keyboard, mouse, scripting, device, customer 
and idle modes; regions of the images 7 which correspond to particular modes 
and any functions that are specific to any particular mode. 

Figure 3 illustrates a flowchart of example operations of a main process 
10 18. In this example, the main process 18 comprises a loop structure. A main 
process 18 is invoked and proceeds to initialize the system 20. A first step in 
the loop is to check for any termination signals 21 that signal arrest of the main 
process 18. An example of a termination signal is the switching off of the power 
supply. 

15 A second step checks for start mode signals 22 which request the 

starting of a sub process in a particular mode. Start_mode signals are not 
typically generated in response to input movements, but are issued in response 
to extrinsic factors such as connection of an ancillary device to the computing 
system or in response to a signal generated by application software installed 

20 on the computing system. By way of alternative, starMnode signals may be 
generated via a manually operable switch providing for selection between 
different modes. If a start_mode signal is detected, the main process 18 will 
invoke a sub process in the mode indicated 23. The invoked sub process 
operates simultaneously to the main process 18. If insufficient central 

25 processing units (CPUs) are available, a standard time sharing mechanism may 
be used to execute multiple processes simultaneously. 

A successfully invoked sub process may send exclusion signals 24 to 
disable other sub processes which operate in conflicting modes. For example, 
a sub process operating in cursor mode may send an exclusion signal to 

30 prevent the input device 1 from concurrently operating in keyboard mode. If a 
sub process has a higher priority value than other sub processes, the sub 
process is able to exclude all sub processes that have a lower priority value. 
When a sub process 23 is invoked and no exclusion signals or termination 
signals are detected 24, the successfully invoked sub process may send a 
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signal to disable sub processes in keyboard mode. If a keyboard disabling 
signal is detected and a keyboard sub process exists, then the keyboard mode 
sub process 26 will be disabled. Similarly, if no keyboard disabling signal is 
detected and the keyboard sub process does not exist, the keyboard mode sub 
5 process is invoked 25 as the default sub process to enable user input. 

The next step of the main process 18 is to process consecutive image 
frames which includes the deployment of known image processing techniques 
to facilitate the recognition of input movements 27. One such approach is to 
convert the images^? into binary form. Binary images are easier to perform 

10 operations on, thereby reducing the level of computation required to speed up 
the process. Comparisons are made between images^? representing an input 
movement pattern and a bank of previously defined images that correspond to 
appropriate instruction signals in the knowledge database 9. A main process 18 
thereby establishes whether the input movement corresponds to a prompt to 

15 invoke any particular mode 28. Where a particular mode is indicated, the main 
process 18 invokes a sub process in that mode 29. If a signal requesting a sub 
process is detected 28, then the corresponding sub process may be invoked. 
For example, an application software might send a request to start a particular 
sub process for its needs. Such a request might be sent by a running sub 

20 process. 

In the event that the input movement detected does not correspond to a 
prompt for a sub process to be invoked, the main process 18 proceeds directly 
to management of the sub processes 30. Management of sub processes 30 
involves the allocation of system resources to all operating sub processes and 
25 monitoring their progress. These functions need only be performed at regular 
intervals, for example, every two frames or every two seconds. 

The final step in the loop is the processing of instruction signals provided 
by any sub processes and making them available for transmission to the 
computing system 31. A main process 18 may continue to repeat the loop 
30 described until a termination signal is detected 21. 

Figure 4 is a flow chart illustrating a typical sub process 19. When a 
main pr-eves process 18 invokes a sub process 19, it must be assigned a priority 
value 32. The assigning of a priority value occurs via a registration process 33 
which assists the main process in management of any operating sub processes. 
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By assigning priority values 32, a sub process 19 having a minor priority value 
is prevented from impeding a sub process 19 having a major priority value. The 
order of the priority value assigned to each sub process 19 is dependent on a 
number of factors including: 
5 (a) how the sub process 19 was invoked, e.g. by a device or 

application software or by detected input movements; 

(b) when the sub process 19 was requested; 

(c) the original position of the input movement which invoked the sub 
process 19 on the contact surface 4; and 

10 (d) the mode requested for the sub process 19. 

Once a sub process 19 has been prioritised, it is registered in a 
registration list 33. The registration list records data on registered sub 
processes 19 including sub process identification, modes, priorities and the 
identification of the main process 18 which initiated the sub process 19. 

15 If registration of a sub process is unsuccessful 34, the sub processes 

cannot proceed. Failure of registration of a sub process 34 may arise because: 

(a) the registration list is full; 

(b) there are insufficient system resources available to execute the 
sub process, e.g. the random access memory (RAM) is full or a request for 

20 allocation of resources from another systems was denied; 

(c) a registered sub process having a higher priority value issues a 
restriction signal affecting the sub process in question; 

(d) there is insufficient stored information about the requested mode 
of operation, e.g. there is no predefined bank of instruction signals 

25 corresponding to the input movements; and 

(e) a particular device mode is requested and connection to the 
particular device is not detected. 

On successful registration 34, a sub process claims "ownership" of a 
region 35 of the sensing panel 3 which corresponds to a region in each image 
30 frame. Once a region is claimed 35, any input movements received via that 
region will be processed by the sub process claiming "ownership". The 
registration process records claimed regions. If a particular region has been 
claimed by a sub process having a certain priority value, only that particular sub 
process or a sub process having a higher priority value can process input 
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movements entered via that region. Unclaimed regions are accessed by all sub 
processes indiscriminately. A claim for a region is successful if the region is 
unclaimed or if it has been previously claimed by a sub process having a lower 
priority than the sub process currently seeking the claim. 
5 Changes in "ownership" occur in one of two ways. The registration list 

may keep track of previous claims to regions of the sensing panel 3 so that 
when a sub process completes processing a particular input movement, the 
region is returned to the immediately previous "owner". For example, if a region 
was claimed by Sub Process 1 and then claimed by Sub Process 2 which has a 

10 higher priority than Sub Process 1 and is subsequently claimed by Sub Process 
3 which has a higher priority value than Sub Process 2. After completion of Sub 
Process 3, the region reverts to Sub Process 2. If Sub Process 2 is completed 
then the region will revert to the next immediately previous "owner" being Sub 
Process 1. Once Sub Process 1 is complete, the region reverts to an 

15 unclaimed state. Alternatively, once a sub process is completed, any claimed 
regions automatically revert to unclaimed status. The region is then available to 
be claimed by a previous "owner" or any other sub process 19. 

More than one sub process 19 may claim the same or part of the same 
region of the sensing panel 3 resulting in an overlap of regions claimed by 

20 independent sub processes. Sub processes claiming overlapping regions of the 
sensing panel 3 proceed on the basis of priority values. 

Once a region has been successfully claimed, the sub process initializes 
assorted variables associated with the sub process 37. The sub process may 
issue restriction signals to enforce priority over other operating sub processes 

25 38. At this point a sub process may be combined with other cooperative sub 
processes in order to process complex input movements 39. 

The next step in the loop checks for termination or restriction signals 
which may arrest the sub process 40. As previously described, detection of a 
termination signal arrests the sub process. If a restriction signal effecting the 

30 sub process is detected, it is suspended until that signal has ceased. 

If the sub process proceeds, i.e. no termination or restriction signals were 
detected; the sub process proceeds to process the regions of the image frames 
corresponding to the claimed areas 41 of the sensing panel 3. Image— 7 
processing involves the deployment of known imaged processing techniques 
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as described for the main process 18. Sub processes in different modes may 
have different approaches to processing image frames. It is to be understood 
that sub processes and main processes may also have different approaches to 
processing image frames in the same mode. 
5 A sub process translates the input movements by searching the 

knowledge database 9 in an attempt to match the detected input movements 
with the bank of stored input movements and corresponding instruction signals 
42. If a corresponding instruction signal is identified, it is provided to the main 
process for further processing, such as transmission to the computing system 

10 43-for output. 

This loop is repeated until a termination signal is detected to arrest the 
sub process 40. A termination signal may arise externally to the system; from 
another sub process; may be derived from the sub process itself; or arise in 
response to failure in claiming a region of the sensing panel 3. 

15 Figure 5 exemplifies an interaction between a main process 44 and two 

sub processes 45,46. The cursor mode sub process 46 has a major priority 
whilst the keyboard mode sub process 45 has a minor priority. The cursor 
mode sub process 46 issues an exclusion signal 54 t o prevent other sub 
processes 4&-from being executed whilst it is functioning. 

20 Figure 5 further includes a time sequence 47 to demonstrate the 

interaction between the processes over time~47. Once the main process 44 
has undergone initialization 48, if no termination signals or restriction signals 
are detected to disable the keyboard mode sub process 45, the main process 
44 automatically invokes the keyboard mode sub process 45 as the default sub 

25 process to enable user input-40. 

The main process 44 operates concurrently to the keyboard mode sub 
process 44. The main process 44 detects input movements on unclaimed 
regions of the sensing panel-S 50 to locate new input movements-§0, whilst the 
sub process 45 operates on a claimed region or regions of the sensing panel 3 

30 to detect input movements designating keystrokes 51 . 

The sub process 45 identifies an input movement which corresponds to 
an instruction signal to input the character "A" 52. Therefore, the character "A" 
is transmitted to the computing system for output. The main process 
simultaneously detects a new input movement which indicates a prompt to 
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invoke a cursor mode sub process 53. Since the cursor mode sub process 46 
has a higher priority than the keyboard mode sub 45, an exclusion signal 54 is 
issued to prevent the keyboard mode sub process-4§ from being executed 
concurrently. Furthermore, the exclusion signal causes the main process 44 to 
5 manage the sub processes 45,46 without continuing to detect new input 
movements on unclaimed regions of the sensing panel-3. 

The cursor mode sub process processes images-^ to identify input 
movements designating a cursor movement 55. The sub process continues 
until a termination signal is detected 56. The termination signal may be provide 

10 via an input movement such as ceasing all contact with the sensing panel-S for 
a certain length of time. Once the cursor mode sub process has been disabled, 
the exclusion signals cease. Therefore, the main process 44 restarts the 
keyboard mode sub process 45. The keyboard mode sub process 45 reverts 
back to operating on the claimed region or regions of the sensing panel-3 to 

15 locate input movements designating keystrokes 57. The sub process identifies 
an input movement corresponding to an instruction signal to input the "Space" 
character 58. Therefore, the "Space" character is transmitted to the computing 
system for output. 

The keyboard mode sub process 45 then detects an input movement 
20 within the claimed region which designates a prompt to invoke the cursor mode 
sub process 59. The cursor mode sub process 46 issues an exclusion signal 
60 to arrest the keyboard mode sub process 45. 

Figure 6 illustrates a variation of the example illustrated in fFigure 5. In 
this example, the cursor mode sub process 46 does not issue an exclusion 
25 signal. Therefore, the cursor mode sub process 46 and the keyboard mode sub 
process 45 may operate simultaneously. However, the cursor mode sub 
process 46 does issue a restriction signals to other cursor mode sub processes 
to enable only one cursor mode sub process 46 to operate at any one time. 
Therefore, when main process 44 or keyboard mode sub process 45 detect 
30 input movements designating a prompt to commence the cursor mode sub 
process-S4, the request is denied_61_. Once the cursor mode sub process 46 
has been terminated-©2, detection of another input movement designating a 
prompt to commence the cursor mode sub process 63 is granted. 
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Figure 7 provides a more complex example, wherein five sub processes 
operate in various modes in conjunction with a main process. In this particular 
example the modes are keyboard, cursor and scripting modes and two device 
modes. Each mode is associated with a sub process 65, 66, 67, 68 and 69. 
5 The keyboard sub process 65 has the lowest priority value and the device 2 sub 
process 69 has the highest priority value. After initialization 70, the main 
process 64 invokes the default keyboard mode sub process 65. The keyboard 
mode sub process 65 detects input movements designating a prompt 71 to 
commence the cursor mode sub process 66. The cursor mode sub process 66 
10 is invoked and issues a restriction signal 72 to arrest other cursor mode sub 
processes or the scripting mode sub process 67. The cursor sub process 66 
continues to operate until a termination signal is detected 73. On arrest of the 
cursor mode sub process 66, the restriction signals inhibiting sub processes 66 
and 67 cease. 

15 Subsequently, the main process 64 detects an input movement 

designating a prompt 74 to commence the scripting mode sub process 67. The 
scripting mode sub process 67 issues a restriction signal 75 to the cursor mode 
sub process 66 and other scripting mode sub processes 67. Therefore, any 
subsequent requests to invoke the cursor mode sub process 66 are denied, 

20 whilst sub process 65 and 67 are executed. 

The main process 64 later detects an input movement in an unclaimed 
region designating a prompt 76 for the device mode for ancillary device 1. 
Since device mode sub process 68 is not compatible with the keyboard mode 
sub process 65 it issues a restriction signal 77 to arrest the keyboard mode sub 

25 process 65. Moreover, the device mode sub process 68 prevents the cursor 
mode sub process 66 and scripting mode sub process 67 from being invoked. 
Later when an input movement designating a prompt 78 for invoking a new 
cursor mode sub process 66 by the main process 64, the request is denied. 
Once device mode sub process 68 is complete 79, the restriction signal ceases. 

30 The main process 64 immediately invokes the default mode keyboard mode sub 
process 65 and cursor mode sub process 66 in response to an appropriate 
prompt. 

The main process 64 subsequently detects a prompt to invoke device 
mode sub process 69 for a second ancillary device. The device mode sub 
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process 69 issues an exclusion signal 80 to arrest any other sub processes 
currently operating. The main process 64 does not detect input movements on 
unclaimed regions to identify input movements designating prompts to invoke 
other sub processes at this time. Therefore, instruction signals are only 
5 generated by device mode sub process 69. Once device mode sub process 69 
is completed 81, the exclusion and restriction signals are ceased. The default 
keyboard mode sub process 65 is invoked 82 by the main process 64. 

Figure 8 illustrates a keyboard layout 11 for use with the present 
invention. The keyboard layout 11 is based on the standard QWERTY layout. 
10 However, it is to be appreciated that a user may modify the keyboard layout 1 1 
to suit individual needs. If modifications in layout are required, the keyboard 
layout 1 1 information stored in the knowledge database 9 may be updated as 
required. Where the sensing panel 3 includes a LCD layer 17 displaying the 
keyboard layout 1 1 , any changes to the keyboard layout 1 1 will be reflected by 
15 the display. If the keyboard layout 11 is printed directly onto a layer of the 
sensing panel 3, it may be reprinted to replace the default arrangement. 

Using the example of the QWERTY keyboard layout 1 1 , at least five 
input movements are recognised. These include PKeyType, PKeyHold, 
PKeyType2, PkeyDoubieType and PKeyDoubleType2. Details of these input 
20 movements are described below. 

Figure 9 exemplifies some user input patterns which may be used to 
provide input to a computing system. 

P0 indicates that no contact has been detected by the sensors 14 within 
the sensing panel 3. 

25 PF1 indicates that a single finger 6 has been positioned on the contact 

surface 4 of the sensing panel 3. The placement of a single finger 6 on the 
contact surface 4 is sensed as a substantially elliptical form or impression 16 
and recorded as such in the image 7. The image 7 records parameters such as 
the moment in time when the contact took place, the duration, size, position and 

30 perhaps colour or intensity of the impression 16. 

. PF2 indicates the detection of an input movement using a single finger 6. 
Detection of the input movement comprises detection of an original finger 6 
position and a current finger 6 position resulting in a displacement vector 
representing the change between the two positions in distance and direction. 
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Other attributes of the movement such as pressure and duration of contact may 
also be recorded. 

PF3 designates removal of a finger 6 from the contact surface 4. This 
input movement is essentially an opposite input movement to that designated 
5 byPF1. 

PF4 designates a finger 6 is positioned on the contact surface 4. The 
duration of the contact is recorded. 

Based on the exemplary basic input movement patterns, more complex 
input movement patterns may be constructed. By means of example, some 
10 more complex input movements are described below. 

Figure 9 further illustrates the following input movements. 

PKeyType is a sequence of PO + PF1 + PF4 + PF3, wherein the 
duration of PF4 is very brief and the recorded positions of PF1 and PF3 are in 
close proximity to one another. This input movement can be equated to 
15 pressing and instantly releasing a key 20 on a conventional keyboard. 

PKeyHold is a sequence of PO + PF1 + PF4. This input movement may 
be equated to pressing and holding down a key 20. 

PKeyType2 is a sequence of PF4 + PF3 + PO + PF1 wherein the 
positions recorded for PF4, PF3 and PF1 are in close proximity and the duration 
20 of PO is very short. This input movement may be equated with removing a 
user's finger 6 from an original position and rapidly returning it to the original 
position. 

Figure 10 exemplifies more input movements which may be used to 
provide input to a computing system for output. 

25 PKeyDoubleType is a sequence of P0 + PF1 + PF4 + PF3 + P0 + PF1 + 

PF1 + PF4 wherein the duration of the first PF4 and the second P0 should be 
very short. This input movement may be otherwise interpreted as a PKeyType 
immediately followed by a PKeyHold, or pressing down a key 20 (see Figure 8) , 
releasing it, and then pressing the key 20 down again and holding it. 

30 PKeyType2 is a sequence of PF4 + PF3 + P0 + PF1 + PF4 + PF3 + P0 + 

PF1 + PF4. The duration of the P0 strokes and the P4 stroke are very short 
and the recorded positions of PF1, PF3 and PF4 are in close proximity. This 
type of sequence may be otherwise interpreted as a PKeyType2 sequence 
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directly following an earlier PKeyType2 sequence, or lifting of a user's finger 6 
twice from a maintained holding position. 

Figure 1 1 exemplifies yet more user input movements which may provide 
input to a computing system in accordance with the present invention. 
5 PCursorRB is a sequence wherein two PO + PF1 sequences occur 

simultaneously and the recorded position of the two PF1s is adjacent to one 
another. This sequence mimics the act of placing two fingers 6 on the contact 
surface 4 side by side. 

PCursorAB is similar to the PCursorRB except that one of the user's 
10 fingers 6 touches the contact surface 4 before the other. That is, the sequence 
of finger 1 is PO + PF1 + PF4 whilst the sequence of finger 2 is PO + PF1 and 
the time difference between the two PF1s corresponds to the duration of PF4. 
The duration should be in some sort of predefined range, for example 0.6 to 2 
seconds. 

15 When a PKeyType input movement is detected in a position 

corresponding to a key 20 position, then the instruction signal transmitted to the 
computing system will be to produce an output of the character associated with 
that particular key 20. If a PKeyHold input movement is detected in a position 
associated with a functional key 20 such as "Ctrl", "Shift" or "Alt", then the 

20 keyboard mode sub process will incorporate the function into the instruction 
signal transmitted to the computing system until the input movement is no 
longer detected. However, if the same input movement is detected on a non 
functional key 20 such as "A", "5" or "Space", it will result in an instruction signal 
being transmitted to the computing system which does not result in any output. 

25 That is, if the users intention is to output a sequence of "A"s, the user must 
perform input movements such as PKeyDoubleType as previously described. A 
keyboard mode sub process then causes an instruction signal to be transmitted 
to the computing system resulting in an output of a sequence of "A"s until such 
time that a new input movement is detected such as lifting a finger 6 (PF3) or 

30 moving a finger 6 (PF2). 

It is to be understood that the association between input movements and 
corresponding instruction signals is dynamic. That it, the knowledge database 9 
may be redefined to adapt the input device 1 to preferences of individual users. 
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As described earlier, a user can prompt initiation of a cursor mode sub 
process by applying more than one finger 6 to the contact surface 4 in a 
PCursorRB or PCursorAB sequence. For instance, once a PCursorRB sub 
process is invoked, during initialization, the original position of the input 
5 movement is recorded. The position of the users fingers 6 on the contact 
surface 4 will be directly reflected by the position of the cursor on a computing 
system screen. For instance, placing the users fingers 6 in a top left hand 
corner of the contact surface 4 results in the cursor being positioned in the top 
left hand corner of the screen. If the resolution of the screen is & x S y pixels, 
10 the resolution of the panel is L x L pixels and the original position of the fingers 
6 on the panel is (X a ,Y a ) and (X b ,Yb), the cursor will appear at position 



PKeyType2 input movements may simulate standard mouse operations 
such as clicking the right button, clicking the left button or double clicking a 

15 button. If no finger 6 contact is detected for any length of time, then a 
termination signal is issued to arrest the cursor mode sub process. 

In another embodiment, a movement indicating device is provided for 
use with the input device 1 of the present invention. The sensing panel 3 may 
detect input movements delivered by the movement indicating device, whose 

20 movement across the surface of the panel designates an instruction signal to 
move a cursor for example, in the direction indicated by the movement 
indicating device. In one arrangement, the movement indicating device may 
take the form of a hemispherical device. The device is preferably substantially 
formed from or layered in a suitable resilient material which will provide 

25 cushioning support and tactile feedback to a users hand. The movement 
indicting device may be used to indicate movement as follows by guiding 
displacement of the movement indicating device on the contact surface 4 with a 
users hand. 



30 the X axis; 

(b) movement of the movement indicating device in the direction of 
the Y axis; 




(a) 



movement of the movement indicating device in the direction of 
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(c) the degree of tilt of the movement indicating device in the direction 
of the X axis; 

(d) the degree of tilt of the movement indicating device in the direction 
of the Y axis; 

5 (e) rotation of the movement indicating device; and 

(f) application of downward pressure on the movement indicating 
device designates an instruction signal to move downwards and reducing the 
pressure applied to the movement indicating device designates an instruction 
signal to move upwards. 

10 It will be appreciated that the movement indicating device described is 

particularly suited to providing input to a computing system for graphics 
applications such as three dimensional object design programs. 

In another arrangement, the movement indicating device may simulate a 
string instrument. In this case, the device includes a frame with at least four 

15 strings. Each string designates a particular note for auditory output. The input 
movements will be the position of the users fingers 6 on the strings. The 
position of a finger 6 along the string indicates the pitch of the note designated 
by the selected string. The duration of contact with the string indicates the 
length of the note. Multiple finger 6 contacts on a single string are disregarded, 

20 only the highest pitch indicated by the input movement will be used to provide 
auditory output. 

It is to be understood that various additions, alterations and/or 
modifications may be made to the parts previously described without departing 
from the ambit of the invention. 

25 
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Abstract 

A method and system are provided for entering input into a computing 

system. The method involves detecting one or multiple input movements using 
a sensing panel associated with the computing system and classifying each 
5 detected input movement as being of a particular type. Each detected input 
movement is translated to an instruction signal by consulting a knowledge 
database. The instruction signals are transmitted to the computing system for 
execution. Translation of a detected input movement to an instruction signal 
involves a main process and one or more sub-processes, wherein each sub 

10 process is invoked by the main process in response to a particular type of 
detected input movement. 

A m e thod for ent e r i ng input into a computing syst e m inc l ud e s th e st e ps 
of: d e t e ct i ng i nput mov e m e nts by m e ans of a pan e l (3) including an array of 
sensors — — processing the d e tect e d — i nput movem e nts by consulting a 

15 knowledg e databas e (9) to i d e ntify a corresponding i nstruction signal for e ach 
detected — input mov e m e nt; and transmitting th e instruct i on s i gna l to th e 
computing syst e m. — Also provid e d i s an input system having a pan el (3) 
includ i ng an array of sensors (5) for d e t e cting input mov e m e nts; a proc e ssor (8) 
for process i ng th e detected i nput movements by consu l t i ng a knowl e dg e 

20 d a tabase (9) to id e ntify a corresponding instruct i on signal for each d e tect e d 
input — movement; — a-nd — a — si gnal transmission — m e ans — for transmitt i ng — the 
instruction signals to th e computing syst e m. 
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